POLA: a student modeling framework for Probabilistic On-Line Assessment of problem solving performance

The paper presents POLA, a student modeling framework that performs probabilistic assessment of students’ performance while they solve problems in introductory Physics. Existing efforts toward probabilistic student modeling focus exclusively on performing knowledge tracing. With POLA we aim to turn OLAE, a system that performs probabilistic knowledge tracing, into a system that applies probabilistic reasoning to perform both knowledge and model tracing. POLA generates probabilistic predictions about the student’s line of reasoning without using heuristics, even when the problem’s solution space is large. An AND/OR graph provides a compact representation of all the available solutions for a problem. A Bayesian network built incrementally from the AND/OR graph and from the student’s actions generates predictions about the solution that the student is following. At the end of the problem solving session the network provides an assessment of student’s level of mastery of the physics knowledge involved in the problem’s solution. Introduction There has been a great deal of interest recently in applying probabilistic methods, especially Bayesian networks, to reasoning about uncertainty (Pearl 1988). The problem of inferring from a tutorial interaction what a student is thinking and what knowledge she has clearly is a problem of uncertain reasoning, so there have been several recent attempts to apply probabilistic reasoning to student modeling (Villano 1992; Martin & VanLehn 1995; Petrushin 1993; Sime 1993; Mislevy 1995; Duncan, Brna, & Morss 1994; Gitomer et al. 1995). This paper discusses a new approach to applying Bayesian networks to student modeling. There are actually two types of student modeling, which Anderson, Corbett and Koedinger (Anderson et al. 1995) called knowledge tracing and model tracing. These are their names for the particular techniques they use, but the distinction is perfectly general. Knowledge tracing refers to the problem of determining what students know, including both correct domain knowledge and robust misconceptions. Model tracing refers to tracking a student’s problem solving as she works on a problem. Model tracing is useful for systems that attempt to answer requests for help or to give unsolicited hints and feedback in the middle of problem solving. In fact, to do an adequate job of helping, hinting and critiquing an on-going solution attempt a system must at minimum understand what line of reasoning the student is attempting to pursue. On the other hand,knowledge tracing is useful for making longer range pedagogical decisions, such as what problem to assign next or what evaluation grade to assign to the student. All the existing attempts to apply probabilistic reasoning to student modeling have been directed toward performing knowledge tracing only. The research project reported in this paper aims to turn an existing system that does probabilistic knowledge tracing, OLAE (Martin & VanLehn 1995), into a system that applies probabilistic reasoning to do both knowledge and model tracing. The new system is called POLA (Probabilistic On-Line Assessment). It will eventually become the student modeling component of a tutoring system that offers help upon request and gives unsolicited hints and feedback during problem solving. The challenge is to get POLA to follow the student’s reasoning without making unwarranted heuristic assumptions. Instead, uncertainty in the interpretation of the student’s actions is to be handled in a sound probabilistic manner. In particular, whenever multiple solutions paths are consistent with the student’s past actions, POLA will be able to assess which one is most probably the one that the student is following. Given such information, the tutoring system can generate reasonable hints and answer help requests. Without it, the tutoring system would be unable to answer even the simplest help request, such as “what should I do next?” Student modeling in domains with large solution spaces Both the OLAE research and this research have been conducted in the task domain of physics (Newtonian mechanics in particular). Physics is somewhat different from other task domains in the student modeling literature in that there are many correct solution paths. For instance, for the problem shown in Figure 1a the 8 primitive equations in (Figure 1b) can be grouped in the two solution sets in Figure 1c, corresponding to two conceptually different solutions, and . Since equations can be generated in any order, each solution can be generated in at least ! different ways, where is the number of N value of the normal force N? a c b PRIMITIVE EQUATIONS mass of boy (Mb) = 75kg weigth of bag of flour (Wf) = 40N Mb = 75 Wf = 40 G = 9.8 Wf = Mf*G Wb = Mb*G Wa = Wf + Wb Ma = Mf + Mb N = Wa where Wa = total weight of boy and flour Ma = total mass of boy and flout Mb = 75 Wf=40 G=9.8 Mf = Mf/G Ma = Mf + Mb Wa = Ma*g N = Wa SOLUTION SET for Mb=75 Wf = 40 G = 9.8 Wb = Mb*G Wa = Wf + Wb N = Wa solution "SummWeight" SOLUTION SET for solution "SummMass" Figure 1: Physics problem with multiple solutions primitive equations in the corresponding solution set. Thus, there are 6! 720 ways to express ! and 7! 5040 ways to express ! . Actually these numbers represent a lower bound for the number of different ways of expressing solutions for this problem because they assume that the student only writes primitive equations. When combinations of primitive equations are written, such as 40